routlier is a package that is built to look for outliers in a dataset. The functions allow a user to look for outliers that are ‘x’ number of deviations away from the mean in the data for a particular column. The number of ‘Outliers’ in a dataset will be returned. Additionally any Outlier value will now be replaced with the word ‘Outlier’ in the dataset.
## You have 66 outliers in your dataset
## FTP UEMP MAN LIC GR CLEAR WM NMAN GOV
## 1 260.35 Outlier Outlier Outlier Outlier 93.4 Outlier Outlier Outlier
## 2 269.8 7 Outlier Outlier Outlier 88.5 Outlier Outlier Outlier
## 3 272.04 5.2 Outlier Outlier Outlier Outlier Outlier Outlier Outlier
## 4 272.96 4.3 535.8 222.1 Outlier 92 500457 591 150.3
## 5 272.51 3.5 576 301.92 297.65 91 482418 626.1 164.3
## 6 261.34 Outlier 601.7 391.22 367.62 87.4 465029 659.8 179.5
## 7 268.89 4.1 577.3 665.56 616.54 88.3 448267 686.2 187.5
## 8 295.99 3.9 596.9 Outlier Outlier 86.1 432109 699.6 195.4
## 9 319.87 3.6 Outlier 837.6 786.23 79 416533 729.9 210.3
## 10 341.43 7.1 569.3 794.9 713.77 73.9 401518 757.8 Outlier
## 11 Outlier Outlier 548.8 817.74 750.43 Outlier Outlier 755.3 Outlier
## 12 Outlier 7.7 563.4 583.17 Outlier Outlier Outlier Outlier Outlier
## 13 Outlier 6.3 Outlier 709.59 666.5 Outlier Outlier Outlier Outlier
## HE WE HOM ACC ASR
## 1 Outlier Outlier Outlier Outlier 306.18
## 2 3.09 134.02 8.9 Outlier 315.16
## 3 3.23 141.68 Outlier 45.31 277.53
## 4 3.33 147.98 8.89 49.51 Outlier
## 5 3.46 159.85 13.07 Outlier Outlier
## 6 3.6 157.19 14.57 Outlier Outlier
## 7 3.73 155.29 21.36 50.62 286.11
## 8 Outlier 131.75 28.03 51.47 291.59
## 9 4.25 178.74 31.49 49.16 320.39
## 10 4.47 178.3 37.39 45.8 323.03
## 11 Outlier 209.54 Outlier 44.54 357.38
## 12 Outlier Outlier Outlier Outlier Outlier
## 13 Outlier Outlier Outlier 44.17 Outlier
Here we will utilize the student dataset that is included in the routlier package. This dataset has both quantitative and qualitative data in it. You can see we have 274 outliers when we set the sd argument equal to 2.
## You have 274 outliers in your dataset
## age Medu Fedu traveltime studytime failures famrel freetime
## 1 18 4 4 2 2 0 4 3
## 2 17 1 1 1 2 0 5 3
## 3 15 1 1 1 2 Outlier 4 3
## 4 15 4 2 1 3 0 3 2
## 5 16 3 3 1 2 0 4 3
## 6 16 4 3 1 2 0 5 4
## 7 16 2 2 1 2 0 4 4
## 8 17 4 4 2 2 0 4 Outlier
## 9 15 3 2 1 2 0 4 2
## 10 15 3 4 1 2 0 5 5
## 11 15 4 4 1 2 0 3 3
## 12 15 2 1 Outlier 3 0 5 2
## 13 15 4 4 1 1 0 4 3
## 14 15 4 3 2 2 0 5 4
## 15 15 2 2 1 3 0 4 5
...
## You have 66 outliers in your dataset
## You have 66 outliers in your dataset
## You have 4 outliers in your dataset
## [1] "WM" "WE" "UEMP" "NMAN" "MAN" "LIC" "HOM" "HE" "GR"
## [10] "GOV" "FTP" "CLEAR" "ASR" "ACC"
## [1] "WM" "WE" "UEMP" "NMAN" "MAN" "LIC" "HOM" "HE" "GR"
## [10] "GOV" "FTP" "CLEAR" "ASR" "ACC"
| WM | WE | UEMP | NMAN | MAN | LIC | HOM | HE | GR | GOV | FTP | CLEAR | ASR | ACC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 558724.00 | 117.18 | 11.00 | 538.10 | 455.50 | 178.50 | 8.60 | 2.98 | 215.98 | 133.90 | 260.35 | 93.40 | 306.18 | 39.17 |
| 538584.00 | 134.02 | 7.00 | 547.60 | 480.20 | 156.41 | 8.90 | 3.09 | 180.48 | 137.60 | 269.80 | 88.50 | 315.16 | 40.27 |
| 519171.00 | 141.68 | 5.20 | 562.80 | 506.10 | 198.02 | 8.52 | 3.23 | 209.57 | 143.60 | 272.04 | 94.40 | 277.53 | 45.31 |
| 500457.00 | 147.98 | 4.30 | 591.00 | 535.80 | 222.10 | 8.89 | 3.33 | 231.67 | 150.30 | 272.96 | 92.00 | 234.07 | 49.51 |
| 482418.00 | 159.85 | 3.50 | 626.10 | 576.00 | 301.92 | 13.07 | 3.46 | 297.65 | 164.30 | 272.51 | 91.00 | 230.84 | 55.05 |
| 465029.00 | 157.19 | 3.20 | 659.80 | 601.70 | 391.22 | 14.57 | 3.60 | 367.62 | 179.50 | 261.34 | 87.40 | 217.99 | 53.90 |
| 448267.00 | 155.29 | 4.10 | 686.20 | 577.30 | 665.56 | 21.36 | 3.73 | 616.54 | 187.50 | 268.89 | 88.30 | 286.11 | 50.62 |
| 432109.00 | 131.75 | 3.90 | 699.60 | 596.90 | 1131.21 | 28.03 | 2.91 | 1029.75 | 195.40 | 295.99 | 86.10 | 291.59 | 51.47 |
| 416533.00 | 178.74 | 3.60 | 729.90 | 613.50 | 837.60 | 31.49 | 4.25 | 786.23 | 210.30 | 319.87 | 79.00 | 320.39 | 49.16 |
| 401518.00 | 178.30 | 7.10 | 757.80 | 569.30 | 794.90 | 37.39 | 4.47 | 713.77 | 223.80 | 341.43 | 73.90 | 323.03 | 45.80 |
| 387046.00 | 209.54 | 8.40 | 755.30 | 548.80 | 817.74 | 46.26 | 5.04 | 750.43 | 227.70 | 356.59 | 63.40 | 357.38 | 44.54 |
| 373095.00 | 240.05 | 7.70 | 787.00 | 563.40 | 583.17 | 47.24 | 5.47 | 1027.38 | 230.90 | 376.69 | 62.50 | 422.07 | 41.03 |
| 359647.00 | 258.05 | 6.30 | 819.80 | 609.30 | 709.59 | 52.33 | 5.76 | 666.50 | 230.20 | 390.19 | 58.90 | 473.01 | 44.17 |
## [1] "The MAD for column 1 is from: 329.046758 : 216.873242 and the overall MAD range is: 112.173516"
## [1] "The MAD for column 2 is from: 12.76126 : -2.36126 and the overall MAD range is: 15.12252"
## [1] "The MAD for column 3 is from: 713.40872 : 425.19128 and the overall MAD range is: 288.217440000001"
## [1] "The MAD for column 4 is from: 1714.823754 : -548.483754 and the overall MAD range is: 2263.307508"
## [1] "The MAD for column 5 is from: 2034.898942 : -801.818942 and the overall MAD range is: 2836.717884"
## [1] "The MAD for column 6 is from: 114.0868 : 60.7132 and the overall MAD range is: 53.3736"
## [1] "The MAD for column 7 is from: 680397.682 : 216136.318 and the overall MAD range is: 464261.364"
## [1] "The MAD for column 8 is from: 1004.66248 : 367.73752 and the overall MAD range is: 636.924959999999"
## [1] "The MAD for column 9 is from: 352.95816 : 22.0418400000001 and the overall MAD range is: 330.91632"
## [1] "The MAD for column 10 is from: 6.357636 : 0.842363999999999 and the overall MAD range is: 5.515272"
## [1] "The MAD for column 11 is from: 253.04009 : 61.3399099999999 and the overall MAD range is: 191.70018"
## [1] "The MAD for column 12 is from: 76.824066 : -34.104066 and the overall MAD range is: 110.928132"
## [1] "The MAD for column 13 is from: 67.016006 : 24.583994 and the overall MAD range is: 42.432012"
## [1] "The MAD for column 14 is from: 433.60947 : 178.75053 and the overall MAD range is: 254.85894"
## [1] "You have a total of 7 Outliers in your dataset"
| FTP | UEMP | MAN | LIC | GR | CLEAR | WM | NMAN | GOV | HE | WE | HOM | ACC | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 260.35 | 11.00 | 455.50 | 178.50 | 215.98 | 93.40 | 558724.00 | 538.10 | 133.90 | 2.98 | 117.18 | 8.60 | 39.17 | 306.18 |
| 269.80 | 7.00 | 480.20 | 156.41 | 180.48 | 88.50 | 538584.00 | 547.60 | 137.60 | 3.09 | 134.02 | 8.90 | 40.27 | 315.16 |
| 272.04 | 5.20 | 506.10 | 198.02 | 209.57 | 94.40 | 519171.00 | 562.80 | 143.60 | 3.23 | 141.68 | 8.52 | 45.31 | 277.53 |
| 272.96 | 4.30 | 535.80 | 222.10 | 231.67 | 92.00 | 500457.00 | 591.00 | 150.30 | 3.33 | 147.98 | 8.89 | 49.51 | 234.07 |
| 272.51 | 3.50 | 576.00 | 301.92 | 297.65 | 91.00 | 482418.00 | 626.10 | 164.30 | 3.46 | 159.85 | 13.07 | 55.05 | 230.84 |
| 261.34 | 3.20 | 601.70 | 391.22 | 367.62 | 87.40 | 465029.00 | 659.80 | 179.50 | 3.60 | 157.19 | 14.57 | 53.90 | 217.99 |
| 268.89 | 4.10 | 577.30 | 665.56 | 616.54 | 88.30 | 448267.00 | 686.20 | 187.50 | 3.73 | 155.29 | 21.36 | 50.62 | 286.11 |
| 295.99 | 3.90 | 596.90 | 1131.21 | 1029.75 | 86.10 | 432109.00 | 699.60 | 195.40 | 2.91 | 131.75 | 28.03 | 51.47 | 291.59 |
| 319.87 | 3.60 | 613.50 | 837.60 | 786.23 | 79.00 | 416533.00 | 729.90 | 210.30 | 4.25 | 178.74 | 31.49 | 49.16 | 320.39 |
| 341.43 | 7.10 | 569.30 | 794.90 | 713.77 | 73.90 | 401518.00 | 757.80 | 223.80 | 4.47 | 178.30 | 37.39 | 45.80 | 323.03 |
| 356.59 | 8.40 | 548.80 | 817.74 | 750.43 | 63.40 | 387046.00 | 755.30 | 227.70 | 5.04 | 209.54 | 46.26 | 44.54 | 357.38 |
| 376.69 | 7.70 | 563.40 | 583.17 | 1027.38 | 62.50 | 373095.00 | 787.00 | 230.90 | 5.47 | 240.05 | 47.24 | 41.03 | 422.07 |
| 390.19 | 6.30 | 609.30 | 709.59 | 666.50 | 58.90 | 359647.00 | 819.80 | 230.20 | 5.76 | 258.05 | 52.33 | 44.17 | 473.01 |
This approach utilizes the Tukey method by looking at the quantile ranges utilizing both the upper quartile and lowe quartile ranges.
## [1] "The IQR for column 1 is from: 27.8875 : 13.3875 and the overall IQR range is: 14.5"
## [1] "The IQR for column 2 is from: 8.25 : 2.25 and the overall IQR range is: 6"
## [1] "The IQR for column 3 is from: 399.3375 : 0.437499999999943 and the overall IQR range is: 398.9"
## [1] "The IQR for column 4 is from: 153.125 : 64.125 and the overall IQR range is: 89"
## [1] "The IQR for column 5 is from: 5.0025 : 2.0625 and the overall IQR range is: 2.94"
## [1] "The IQR for column 6 is from: 4.184375 : 2.199375 and the overall IQR range is: 1.985"
## [1] "The IQR for column 7 is from: 24.12 : 12.76 and the overall IQR range is: 11.36"
## [1] "The IQR for column 8 is from: 2.5 : -1.5 and the overall IQR range is: 4"
## [1] "The IQR for column 9 is from: 1.875 : -1.125 and the overall IQR range is: 3"
## [1] "The IQR for column 10 is from: 5.5 : 1.5 and the overall IQR range is: 4"
## [1] "The IQR for column 11 is from: 8.125 : -2.875 and the overall IQR range is: 11"
## [1] "You have a total of 3 Outliers in your dataset"
| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Mazda RX4 | 21.000 | 6.000 | 160.000 | 110.000 | 3.900 | 2.620 | 16.460 | 0.000 | 1.000 | 4.000 | 4.000 |
| Mazda RX4 Wag | 21.000 | 6.000 | 160.000 | 110.000 | 3.900 | 2.875 | 17.020 | 0.000 | 1.000 | 4.000 | 4.000 |
| Datsun 710 | 22.800 | 4.000 | 108.000 | 93.000 | 3.850 | 2.320 | 18.610 | 1.000 | 1.000 | 4.000 | 1.000 |
| Hornet 4 Drive | 21.400 | 6.000 | 258.000 | 110.000 | 3.080 | 3.215 | 19.440 | 1.000 | 0.000 | 3.000 | 1.000 |
| Hornet Sportabout | 18.700 | 8.000 | 360.000 | 175.000 | 3.150 | 3.440 | 17.020 | 0.000 | 0.000 | 3.000 | 2.000 |
| Valiant | 18.100 | 6.000 | 225.000 | 105.000 | 2.760 | 3.460 | 20.220 | 1.000 | 0.000 | 3.000 | 1.000 |
| Duster 360 | 14.300 | 8.000 | 360.000 | 245.000 | 3.210 | 3.570 | 15.840 | 0.000 | 0.000 | 3.000 | 4.000 |
| Merc 240D | 24.400 | 4.000 | 146.700 | 62.000 | 3.690 | 3.190 | 20.000 | 1.000 | 0.000 | 4.000 | 2.000 |
| Merc 230 | 22.800 | 4.000 | 140.800 | 95.000 | 3.920 | 3.150 | 22.900 | 1.000 | 0.000 | 4.000 | 2.000 |
| Merc 280 | 19.200 | 6.000 | 167.600 | 123.000 | 3.920 | 3.440 | 18.300 | 1.000 | 0.000 | 4.000 | 4.000 |
This is the data set called `DETROIT’ in the book ‘Subset selection in regression’ by Alan J. Miller published in the Chapman & Hall series of monographs on Statistics & Applied Probability, no. 40. The data are unusual in that a subset of three predictors can be found which gives a very much better fit to the data than the subsets found from the Efroymson stepwise algorithm, or from forward selection or backward elimination. The original data were given in appendix A of ``Regression analysis and its application: A data-oriented approach’ by Gunst & Mason, Statistics textbooks and monographs no. 24, Marcel Dekker. It has caused problems because some copies of the Gunst & Mason book do not contain all of the data, and because Miller does not say which variables he used as predictors and which is the dependent variable. (HOM was the dependent variable, and the predictors were FTP … WE)
A data frame with 13 rows and 14 variables:
- Detroit Dataset:
- FTP: Full-time police per 100,000 population
- UEMP: UEMP - % unemployed in the population
- MAN: MAN - number of manufacturing workers in thousands
- LIC: LIC - Number of handgun licences per 100,000 population
- GR: GR - Number of handgun registrations per 100,000 population
- CLEAR: CLEAR - % homicides cleared by arrests
- WM: WM - Number of white males in the population
- NMAN: NMAN - Number of non-manufacturing workers in thousands
- GOV: GOV - Number of government workers in thousands
- HE: HE - Average hourly earnings
- WE: WE - Average weekly earnings:
- HOM: HOM - Number of homicides per 100,000 of population
- ACC: ACC - Death rate in accidents per 100,000 population
- ASR: ASR - Number of assaults per 100,000 population
This data approach student achievement in secondary education of two Portuguese schools. The data attributes include student grades, demographic, social and school related features) and it was collected by using school reports and questionnaires. Two datasets are provided regarding the performance in two distinct subjects: Mathematics (mat) and Portuguese language (por). In [Cortez and Silva, 2008], the two datasets were modeled under binary/five-level classification and regression tasks. Important note: the target attribute G3 has a strong correlation with attributes G2 and G1. This occurs because G3 is the final year grade (issued at the 3rd period), while G1 and G2 correspond to the 1st and 2nd period grades. It is more difficult to predict G3 without G2 and G1, but such prediction is much more useful (see paper source for more details)
A data frame with 392 rows and 33 variables:
Attributes for both student-mat.csv (Math course) and student-por.csv (Portuguese language course) datasets:
- school - student’s school (binary: ‘GP’ - Gabriel Pereira or ‘MS’ - Mousinho da Silveira)
- sex - student’s sex (binary: ‘F’ - female or ‘M’ - male)
- age - student’s age (numeric: from 15 to 22)
- address - student’s home address type (binary: ‘U’ - urban or ‘R’ - rural)
- famsize - family size (binary: ‘LE3’ - less or equal to 3 or ‘GT3’ - greater than 3)
- Pstatus - parent’s cohabitation status (binary: ‘T’ - living together or ‘A’ - apart)
- Medu - mother’s education (numeric: 0 - none, 1 - primary education (4th grade), 2 = 5th to 9th grade, 3 = secondary education or 4 = higher education)
- Fedu - father’s education (numeric: 0 - none, 1 - primary education (4th grade), 2 = 5th to 9th grade, 3 = secondary education or 4 = higher education)
- Mjob - mother’s job (nominal: ‘teacher’, ‘health’ care related, civil ‘services’ (e.g. administrative or police), ‘at_home’ or ‘other’)
- Fjob - father’s job (nominal: ‘teacher’, ‘health’ care related, civil ‘services’ (e.g. administrative or police), ‘at_home’ or ‘other’)
- reason - reason to choose this school (nominal: close to ‘home’, school ‘reputation’, ‘course’ preference or ‘other’)
- guardian - student’s guardian (nominal: ‘mother’, ‘father’ or ‘other’)
- traveltime - home to school travel time (numeric: 1 - <15 min., 2 - 15 to 30 min., 3 - 30 min. to 1 hour, or 4 - >1 hour)
- studytime - weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3 - 5 to 10 hours, or 4 - >10 hours)
- failures - number of past class failures (numeric: n if 1<=n<3, else 4)
- schoolsup - extra educational support (binary: yes or no)
- famsup - family educational support (binary: yes or no)
- paid - extra paid classes within the course subject (Math or Portuguese) (binary: yes or no)
- activities - extra-curricular activities (binary: yes or no)
- nursery - attended nursery school (binary: yes or no)
- higher - wants to take higher education (binary: yes or no)
- internet - Internet access at home (binary: yes or no)
- romantic - with a romantic relationship (binary: yes or no)
- famrel - quality of family relationships (numeric: from 1 - very bad to 5 - excellent)
- freetime - free time after school (numeric: from 1 - very low to 5 - very high)
- goout - going out with friends (numeric: from 1 - very low to 5 - very high)
- Dalc - workday alcohol consumption (numeric: from 1 - very low to 5 - very high)
- Walc - weekend alcohol consumption (numeric: from 1 - very low to 5 - very high)
- health - current health status (numeric: from 1 - very bad to 5 - very good)
- absences - number of school absences (numeric: from 0 to 93)
- G1 - first period grade (numeric: from 0 to 20)
- G2 - second period grade (numeric: from 0 to 20)
- G3 - final grade (numeric: from 0 to 20, output target)
Citation Request: Please include this citation if you plan to use this database: P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7. Available at:
Relevant Papers:
P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7. Available at: